This application claims the priority of Japanese Patent Application No. H9-226959 filed on Jul. 18, 1997, which is incorporated herein by reference.
The present invention relates to a motion data preparation system (so-called motion capture system) for detecting a motion locus of a color marker formed on a subject and preparing motion data of the subject based on the detection information.
In recent years, in the field of the movies using animation and the virtual reality, or the movies using computer graphics (CG), techniques for realizing more real motions of animation by inputting actual man""s motion data to synthesized animation characters have been becoming widespread. In this case, the so-called motion capture system in which motions performed by a man equipped with many sensors from head to foot are captured by camera and motion data are prepared by analyzing the captured motions of the sensors on a dynamic image is used. Applying such data to characters prepared by CG or the like makes it possible to represent, with real motions, even a scene that, for example, an actually non-existing monster and men grapple with each other.
Data input in such a motion capture system is implemented by, for example, a process of three-dimensionally measuring spatial positions and twist directions with a mounted magnetic sensor. More specifically, while a receivers (sensor) containing an orthogonal coil is attached to a moving body, an AC current is passed through an orthogonal coil set on the transmitter, where in a magnetic field thereby generated, the body is put into motion. Then, an induced current occurs to the receiver, so that data such as positions and angles of two-dimensional or three-dimensional coordinates by measuring the induced current.
Further, as another method, there is a technique in which a reflecting type target (marker) mounted to a moving body, the moving body is photographed by a plurality of cameras so that the motions of the marker are captured as an image. This technique may be exercised in a method that positional coordinates on the screen are obtained by using as the marker a light-emitting diode that blinks in synchronization with an image signal, or in another method that with the use of a marker that emits light by a special illumination, positional coordinates of the light-emitting body are obtained by image analysis. However, the former method has problems that the motion of the body is limited by the signal cable connecting to the light-emitting diode or that it is difficult to apply this method to small, lightweight bodies. The latter method, on the other hand, is unable to discriminate a plurality of markers from one another, so that additional input operations to a computer would be required, as a disadvantage.
Thus, in order to solve these and other problems, there has been proposed a motion capture system having a constitution that with individual targets attached to a body in the form of color marks, these targets are shot by a video camera while information signals as to color and configuration of the markers are extracted to a video board, where for example centroidal positions are determined based on the extracted information signals, thus allowing two-dimensional or three-dimensional motions of the measurement-object site to be known (see, for example, Japanese Patent Laid-Open Publications H4-93704 and H6-50983).
For image analysis using computers an analysis method using the three elements of red (R), green (G) and blue (B) that are the three primary colors, where their intensities are utilized, is in common use. This method, however, has a difficulty in analysis because the color is represented in three-dimensional space, as well as a problem that errors in data extraction accuracy may occur under influences of changes in illumination because luminance is included as a factor. Further, in a motion capture system that handles color images, because the amount of information on color image data is incommensurably larger than that of gray scale images, memory of quite high capacity and CPU of high speed would be required so that the system would result in increased scale and higher price. Thus, it problematic to introduce this motion capture system. In particular, in an attempt to introduce a monitoring system that uses a plurality of monitor cameras for color images, the required amount of memory would be enormous so that the system""s problems of complexity and high price would be more noticeable, resulting in a bottleneck for widespread use of the system.
An object of the present invention is therefore to provide a motion data preparation system which is enabled, for recognition of color markers provided on a subject body by shooting the color markers with color cameras, to detect the color markers without being affected by the surrounding brightness and thereby acquire stable motion data, and which can execute high-speed, accurate processing by reducing the amount of information on color image data.
In order to achieve the above object, in a first aspect of the present invention, there is provided a motion data preparation system comprising:
one or a plurality of color cameras for shooting motions of a subject in a state that color markers are attached to the subject or that part of the subject is determined as a color marker;
color-integrated image data generating means for, through a decoding process of color image signals based on image outputs from the color cameras, integrating and consolidating color information included in each of the image signals into a specific number of color groups, and thereby generating image data (hereinafter, referred to as color-integrated image data) having data pieces smaller in number than the color image signals;
marker specified-color extracting means for extracting data of pixels having a color previously specified as a color of the color marker (hereinafter, referred to as marker specified color) with respect to the generated color-integrated image data; and
marker-position-data calculating and outputting means for calculating a color marker position of the shot subject based on the extracted marker specified-color pixel data and outputting, as motion data of the subject, marker position data obtained as a result of the calculation.
With this systems image information about color markers on the subject can be dramatically reduced in amount by integrating and consolidating enormous color information of, for example, RGB image signals into a specified number of color groups. Thus, the whole system can be simplified in construction.
The above system may comprise information separating means for, through a process of decoding the color image signal, separating the color image signal into luminance information, which is information on luminance, and chromatic information, which is information on chromatic components. In this cage, the color-integrated image data generating means may be so arranged as to generate the color-integrated image data by integrating and consolidating mainly the separated (or separated and corrected) chromatic information into a specified number of color groups. As a result, the system construction less affected by variations in quantity of light can be realized in addition to the aforementioned simplified system construction.
Also, the system may further comprise image signal digital converting means for converting the color image signal into digital form, and image data generating means for generating image data of the subject by fetching the digitally converted color image signal in specified time intervals. In this case, the information separating means separates chromatic information of pixels of the image data into luminance information comprising a single component, and chromatic information comprising two components of a first chromatic component and a second chromatic component, and the color-integrated image data generating means comprises a color-integration look-up table memory which has stored a correspondence relationship between combinations of first chromatic component information groups and second thromatic component information groups obtained by dividing the first chromatic component information and the second chromatic component information into a specified number of groups, respectively, and chromatic information after the integration (hereinafter, referred to as integrated chromatic information), wherein the color-integrated image data generating means reads, with respect to each pixel, the integrated chromatic information to which the first chromatic component and the second chromatic component of the pixel correspond, from the color-integration look-up table memory in a dictionary-like manner, and assigning the reading result as chromatic data of the pixel.
That is, in order to prepare motion data of a moving subject, image data comprising a set of many pieces of data need to be prepared to many frames, and moreover a great many times of arithmetic operations need to be done for the identification of color marker image portions and the calculation of their marker position data. Further, to obtain motion images of real motion, there is a need of increasing the number of frames per unit time, which causes the number of calculations for marker position data to also increase. Then, with the above constitution, the integration process of two chromatic components of color image data can be promptly achieved by using the look-up table (which hereinafter may be abbreviated as LUT) memory so that the amount of data involved in the calculations can be greatly reduced. Thus, fast motion picture processing can be achieved with simple circuit construction.
More specifically, the chromatic information correcting means may comprise a correction look-up table memory which has stored a correspondence relationship between combinations of integrated chromatic information groups and luminance information groups obtained by dividing the integrated chromatic information and the luminance information into a specified number of groups, respectively and chromatic information after the luminance correction, wherein the chromatic information correcting means reads, with respect to each pixel, chromatic information after the luminance correction to which the integrated chromatic information and the luminance information of the pixel correspond, from the correction look-up table memory in a dictionary-like manner, as corrected color data of the pixel. With this constitution, the reduction in information amount can be achieved more effectively by performing the correction in such a way that luminance information is incorporated into the integrated chromatic information with the use of the correction look-up table memory, and moreover the system is less affected by variations in quantity of light. Thus, the marker position data can be generated with higher precision.
In the above constitution, incorporation (correction) of luminance information is done after the integration of two kinds of chromatic information However, this sequence may be inverted, i.e., the two kinds of chromatic information may be integrated after being corrected with luminance. More specifically, the chromatic information correcting means comprises first and second correction look-up table memories which have stored correspondence relationships between combinations of integrated chromatic information groups and luminance information groups obtained by dividing the chromatic component information and the luminance information into specified numbers of groups in correspondence to the first chromatic component and the second chromatic component, respectively, and chromatic information after the luminance correction, wherein the chromatic information correcting means reads, with respect to each pixel, chromatic information after the luminance correction to which chromatic information and luminance information of the pixel correspond, from their corresponding correction look-up table memories, respectively, in a dictionary-like manner, as corrected chromatic information of the pixel. Further, the color-integrated image data generating means uses the first chromatic information and the second chromatic information those after the luminance correction.
Next, when the system further comprises image signal digital converting means for converting the color image signal into digital form, and image data generating means for generating image data of the subject by fetching the digitally converted color image signal in specified time intervals, the marker-position-data calculating and outputting means may perform calculation and output of color-marker position data based on the marker specified-color pixel data successively and synchronously with the transfer period of the image data (e.g., the period of transfer clock pulse of pixels), for each of the image data. With this constitution, because the calculation of marker position data of each image frame can be executed synchronously with the transfer period of pixels (where, for example, the frequency of transfer clock pulse is about 8-30 MHz) with high speed in a pipeline fashion, the preparation of motion data including the marker position data for each frame can be achieved in real time during the image shooting. In particular, by adopting a clock pulse frequency of 10 MHz (e.g., 14-15 MHz), a high-speed processing environment particularly suitable for real-time preparation of motion data can be realized.
Generally, color image data are given in enormous information amounts as much as 28xc3x9728xc3x9728=16770 thousands when each color component is described as 8-bit (2.56-step) data in the RGB three primary color representation, making it almost impossible to execute real time operations with the processing power of low-priced CPUs used in personal computers or the like. However, reducing the information amount by the integration and consolidation of color information as in the system of the present invention enables the real time operations even with low-priced CPUs (computers) of relatively low processing speed.
In this case, the marker-position-data calculating and outputting means may be so arranged as to calculate and output, for the frame of each of image data, color-marker position data including a total sum xcexa3x of horizontal coordinate components x and a total sum xcexa3y of vertical coordinate components y of pixels included in the marker image area, and a number n of pixels in each area. With this constitution, the centroidal position xcex3 of the image for a color marker can be easily determined by (xcexa3x/n, xcexa3y/n). The centroidal position is not affected by the rotational movement of color markers and therefore easy to manage as motion data and useful for motion image processing.
Furthermore, there is an advantage that xcexa3x and xcexa3y can be simply calculated by utilizing the image synchronization control unit that is necessarily provided, for example, in video image pickup equipment using the normal scanning system. That is, the image signal from the image pickup device is decomposed into pixels, delimited into the individual scanning lines and transferred sequentially. During this transfer, normally, horizontal and vertical synchronizing signals are inserted in the pixel data train. Therefore, then the horizontal coordinate of each pixel on the screen can be specifically determined by counting the number of transferred pixels since the reception of the horizontal synchronizing signal on the receiver side. It is needless to say that the counting of transferred pixels can be done, of course, by using the pixel transfer clock pulse and therefore promptly processed. Meanwhile, by counting the number of scanning lines (equivalent, erg., to the number of horizontal synchronizing signals inserted every one scanning line) since the reception of the vertical synchronizing signal, the vertical coordinate can also be specifically determined. Further, because all of xcexa3x, xcexa3y, and n are simple summed-up amounts, fast processing can be achieved with an extremely simple circuit construction using adders, thus advantageous also in executing the real time operations.
In addition, when smaller numbers of colors are contained in the original image signal, the calculation processing using xcexa3x, xcexa3y and n can be simply implemented by a system constitution as shown below. That is, the system comprises:
one or a plurality of color cameras for shooting motions of a subject in a state that color markers are attached to the subject or that part of the subject is determined as a color marker;
image data generating means for successively generating image data of the subject on a frame basis by fetching color image signals based on image outputs from the color cameras in specified time intervals;
marker specified-color pixel extracting means for extracting data of pixels having a color previously specified as a color of the color marker (hereinafter, referred to as marker specified color) with respect to the successively generated image data; and
marker-position-data calculating and outputting means for, on an assumption that a set of pixels having the marker specified color are taken as a marker image area, calculating marker position information including a total sum xcexa3x of horizontal coordinate components and a total sum xcexa3y of vertical coordinate components y of pixels contained in the marker image area as well as a total number n of pixels in each area, successively in synchronization with a transfer period of the pixels, with respect to each frame of each of the image data.
Next, each of the look-up table memories used in the above constitutions may be so arranged as to designate read-object data specifying information, such as the first and second chromatic component information, the luminance information or the integrated chromatic information, by means of an address line, so that read-object data specified by the read-object data specifying information is read from the memory cell designated by the address in a dictionary-like manner. Also, the system may further comprise read control means serving for making the process of reading the read-object data from the look-up table memory for each of image data carried out successively in synchronization with a transfer period of the pixels. By adopting the look-up table memories having the above constitution, read-object data can be fast accessed by designating the read-object data specifying information by the address line. As a result, high-speed processing synchronized with the pixel transfer period can be realized so that the aforementioned real time processing is enabled without any difficulties
In addition, the color camera may be provided in a plurality. In this case, the system may further comprise a switch to which the plurality of color cameras are connected, and by which cameras with which the subject is shot are selected by switching a state of connection to the cameras. With this arrangement, it becomes possible to generate two-dimensional-like motion data. Moreover, even when one color marker has come to a dead angle, another color camera is enabled to detect this reliably. Thus, the precision for generated marker position data can be improved.